Rethinking the Pipeline of Demosaicing, Denoising and Super-Resolution

Incomplete color sampling, noise degradation, and limited resolution are the three key problems that are unavoidable in modern camera systems. Demosaicing (DM), denoising (DN), and super-resolution (SR) are core components in a digital image processing pipeline to overcome the three problems above, respectively. Although each of these problems has been studied actively, the mixture problem of DM, DN, and SR, which is a higher practical value, lacks enough attention. Such a mixture problem is usually solved by a sequential solution (applying each method independently in a fixed order: DM $\to$ DN $\to$ SR), or is simply tackled by an end-to-end network without enough analysis into interactions among tasks, resulting in an undesired performance drop in the final image quality. In this paper, we rethink the mixture problem from a holistic perspective and propose a new image processing pipeline: DN $\to$ SR $\to$ DM. Extensive experiments show that simply modifying the usual sequential solution by leveraging our proposed pipeline could enhance the image quality by a large margin. We further adopt the proposed pipeline into an end-to-end network, and present Trinity Enhancement Network (TENet). Quantitative and qualitative experiments demonstrate the superiority of our TENet to the state-of-the-art. Besides, we notice the literature lacks a full color sampled dataset. To this end, we contribute a new high-quality full color sampled real-world dataset, namely PixelShift200. Our experiments show the benefit of the proposed PixelShift200 dataset for raw image processing.Comment: Code is available at: https://github.com/guochengqian/TENe

Towards Efficient SDRTV-to-HDRTV by Learning from Image Formation

Modern displays are capable of rendering video content with high dynamic range (HDR) and wide color gamut (WCG). However, the majority of available resources are still in standard dynamic range (SDR). As a result, there is significant value in transforming existing SDR content into the HDRTV standard. In this paper, we define and analyze the SDRTV-to-HDRTV task by modeling the formation of SDRTV/HDRTV content. Our analysis and observations indicate that a naive end-to-end supervised training pipeline suffers from severe gamut transition errors. To address this issue, we propose a novel three-step solution pipeline called HDRTVNet++, which includes adaptive global color mapping, local enhancement, and highlight refinement. The adaptive global color mapping step uses global statistics as guidance to perform image-adaptive color mapping. A local enhancement network is then deployed to enhance local details. Finally, we combine the two sub-networks above as a generator and achieve highlight consistency through GAN-based joint training. Our method is primarily designed for ultra-high-definition TV content and is therefore effective and lightweight for processing 4K resolution images. We also construct a dataset using HDR videos in the HDR10 standard, named HDRTV1K that contains 1235 and 117 training images and 117 testing images, all in 4K resolution. Besides, we select five metrics to evaluate the results of SDRTV-to-HDRTV algorithms. Our final results demonstrate state-of-the-art performance both quantitatively and visually. The code, model and dataset are available at https://github.com/xiaom233/HDRTVNet-plus.Comment: Extended version of HDRTVNe

Self-Supervised Intensity-Event Stereo Matching

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes in microsecond accuracy with a high dynamic range and low power consumption. Despite these advantages, event cameras cannot be directly applied to computational imaging tasks due to the inability to obtain high-quality intensity and events simultaneously. This paper aims to connect a standalone event camera and a modern intensity camera so that the applications can take advantage of both two sensors. We establish this connection through a multi-modal stereo matching task. We first convert events to a reconstructed image and extend the existing stereo networks to this multi-modality condition. We propose a self-supervised method to train the multi-modal stereo network without using ground truth disparity data. The structure loss calculated on image gradients is used to enable self-supervised learning on such multi-modal data. Exploiting the internal stereo constraint between views with different modalities, we introduce general stereo loss functions, including disparity cross-consistency loss and internal disparity loss, leading to improved performance and robustness compared to existing approaches. The experiments demonstrate the effectiveness of the proposed method, especially the proposed general stereo loss functions, on both synthetic and real datasets. At last, we shed light on employing the aligned events and intensity images in downstream tasks, e.g., video interpolation application.Comment: This paper has been accepted by the Journal of Imaging Science & Technolog

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

Automatic speech recognition can potentially benefit from the lip motion patterns, complementing acoustic speech to improve the overall recognition performance, particularly in noise. In this paper we propose an audio-visual fusion strategy that goes beyond simple feature concatenation and learns to automatically align the two modalities, leading to enhanced representations which increase the recognition accuracy in both clean and noisy conditions. We test our strategy on the TCD-TIMIT and LRS2 datasets, designed for large vocabulary continuous speech recognition, applying three types of noise at different power ratios. We also exploit state of the art Sequence-to-Sequence architectures, showing that our method can be easily integrated. Results show relative improvements from 7% up to 30% on TCD-TIMIT over the acoustic modality alone, depending on the acoustic noise level. We anticipate that the fusion strategy can easily generalise to many other multimodal tasks which involve correlated modalities. Code available online on GitHub: https://github.com/georgesterpu/Sigmedia-AVSRComment: In ICMI'18, October 16-20, 2018, Boulder, CO, USA. Equation (2) corrected on this versio

Complete genome sequence of the extremely acidophilic methanotroph isolate V4, Methylacidiphilum infernorum, a representative of the bacterial phylum Verrucomicrobia

<p>Abstract</p> <p>Background</p> <p>The phylum <it>Verrucomicrobia </it>is a widespread but poorly characterized bacterial clade. Although cultivation-independent approaches detect representatives of this phylum in a wide range of environments, including soils, seawater, hot springs and human gastrointestinal tract, only few have been isolated in pure culture. We have recently reported cultivation and initial characterization of an extremely acidophilic methanotrophic member of the <it>Verrucomicrobia</it>, strain V4, isolated from the Hell's Gate geothermal area in New Zealand. Similar organisms were independently isolated from geothermal systems in Italy and Russia.</p> <p>Results</p> <p>We report the complete genome sequence of strain V4, the first one from a representative of the <it>Verrucomicrobia</it>. Isolate V4, initially named "<it>Methylokorus infernorum</it>" (and recently renamed <it>Methylacidiphilum infernorum</it>) is an autotrophic bacterium with a streamlined genome of ~2.3 Mbp that encodes simple signal transduction pathways and has a limited potential for regulation of gene expression. Central metabolism of <it>M. infernorum </it>was reconstructed almost completely and revealed highly interconnected pathways of autotrophic central metabolism and modifications of C₁-utilization pathways compared to other known methylotrophs. The <it>M. infernorum </it>genome does not encode tubulin, which was previously discovered in bacteria of the genus <it>Prosthecobacter</it>, or close homologs of any other signature eukaryotic proteins. Phylogenetic analysis of ribosomal proteins and RNA polymerase subunits unequivocally supports grouping <it>Planctomycetes</it>, <it>Verrucomicrobia </it>and <it>Chlamydiae </it>into a single clade, the PVC superphylum, despite dramatically different gene content in members of these three groups. Comparative-genomic analysis suggests that evolution of the <it>M. infernorum </it>lineage involved extensive horizontal gene exchange with a variety of bacteria. The genome of <it>M. infernorum </it>shows apparent adaptations for existence under extremely acidic conditions including a major upward shift in the isoelectric points of proteins.</p> <p>Conclusion</p> <p>The results of genome analysis of <it>M. infernorum </it>support the monophyly of the PVC superphylum. <it>M. infernorum </it>possesses a streamlined genome but seems to have acquired numerous genes including those for enzymes of methylotrophic pathways <it>via </it>horizontal gene transfer, in particular, from <it>Proteobacteria</it>.</p> <p>Reviewers</p> <p>This article was reviewed by John A. Fuerst, Ludmila Chistoserdova, and Radhey S. Gupta.</p

Canvass: a crowd-sourced, natural-product screening library for exploring biological space

NCATS thanks Dingyin Tao for assistance with compound characterization. This research was supported by the Intramural Research Program of the National Center for Advancing Translational Sciences, National Institutes of Health (NIH). R.B.A. acknowledges support from NSF (CHE-1665145) and NIH (GM126221). M.K.B. acknowledges support from NIH (5R01GM110131). N.Z.B. thanks support from NIGMS, NIH (R01GM114061). J.K.C. acknowledges support from NSF (CHE-1665331). J.C. acknowledges support from the Fogarty International Center, NIH (TW009872). P.A.C. acknowledges support from the National Cancer Institute (NCI), NIH (R01 CA158275), and the NIH/National Institute of Aging (P01 AG012411). N.K.G. acknowledges support from NSF (CHE-1464898). B.C.G. thanks the support of NSF (RUI: 213569), the Camille and Henry Dreyfus Foundation, and the Arnold and Mabel Beckman Foundation. C.C.H. thanks the start-up funds from the Scripps Institution of Oceanography for support. J.N.J. acknowledges support from NIH (GM 063557, GM 084333). A.D.K. thanks the support from NCI, NIH (P01CA125066). D.G.I.K. acknowledges support from the National Center for Complementary and Integrative Health (1 R01 AT008088) and the Fogarty International Center, NIH (U01 TW00313), and gratefully acknowledges courtesies extended by the Government of Madagascar (Ministere des Eaux et Forets). O.K. thanks NIH (R01GM071779) for financial support. T.J.M. acknowledges support from NIH (GM116952). S.M. acknowledges support from NIH (DA045884-01, DA046487-01, AA026949-01), the Office of the Assistant Secretary of Defense for Health Affairs through the Peer Reviewed Medical Research Program (W81XWH-17-1-0256), and NCI, NIH, through a Cancer Center Support Grant (P30 CA008748). K.N.M. thanks the California Department of Food and Agriculture Pierce's Disease and Glassy Winged Sharpshooter Board for support. B.T.M. thanks Michael Mullowney for his contribution in the isolation, elucidation, and submission of the compounds in this work. P.N. acknowledges support from NIH (R01 GM111476). L.E.O. acknowledges support from NIH (R01-HL25854, R01-GM30859, R0-1-NS-12389). L.E.B., J.K.S., and J.A.P. thank the NIH (R35 GM-118173, R24 GM-111625) for research support. F.R. thanks the American Lebanese Syrian Associated Charities (ALSAC) for financial support. I.S. thanks the University of Oklahoma Startup funds for support. J.T.S. acknowledges support from ACS PRF (53767-ND1) and NSF (CHE-1414298), and thanks Drs. Kellan N. Lamb and Michael J. Di Maso for their synthetic contribution. B.S. acknowledges support from NIH (CA78747, CA106150, GM114353, GM115575). W.S. acknowledges support from NIGMS, NIH (R15GM116032, P30 GM103450), and thanks the University of Arkansas for startup funds and the Arkansas Biosciences Institute (ABI) for seed money. C.R.J.S. acknowledges support from NIH (R01GM121656). D.S.T. thanks the support of NIH (T32 CA062948-Gudas) and PhRMA Foundation to A.L.V., NIH (P41 GM076267) to D.S.T., and CCSG NIH (P30 CA008748) to C.B. Thompson. R.E.T. acknowledges support from NIGMS, NIH (GM129465). R.J.T. thanks the American Cancer Society (RSG-12-253-01-CDD) and NSF (CHE1361173) for support. D.A.V. thanks the Camille and Henry Dreyfus Foundation, the National Science Foundation (CHE-0353662, CHE-1005253, and CHE-1725142), the Beckman Foundation, the Sherman Fairchild Foundation, the John Stauffer Charitable Trust, and the Christian Scholars Foundation for support. J.W. acknowledges support from the American Cancer Society through the Research Scholar Grant (RSG-13-011-01-CDD). W.M.W.acknowledges support from NIGMS, NIH (GM119426), and NSF (CHE1755698). A.Z. acknowledges support from NSF (CHE-1463819). (Intramural Research Program of the National Center for Advancing Translational Sciences, National Institutes of Health (NIH); CHE-1665145 - NSF; CHE-1665331 - NSF; CHE-1464898 - NSF; RUI: 213569 - NSF; CHE-1414298 - NSF; CHE1361173 - NSF; CHE1755698 - NSF; CHE-1463819 - NSF; GM126221 - NIH; 5R01GM110131 - NIH; GM 063557 - NIH; GM 084333 - NIH; R01GM071779 - NIH; GM116952 - NIH; DA045884-01 - NIH; DA046487-01 - NIH; AA026949-01 - NIH; R01 GM111476 - NIH; R01-HL25854 - NIH; R01-GM30859 - NIH; R0-1-NS-12389 - NIH; R35 GM-118173 - NIH; R24 GM-111625 - NIH; CA78747 - NIH; CA106150 - NIH; GM114353 - NIH; GM115575 - NIH; R01GM121656 - NIH; T32 CA062948-Gudas - NIH; P41 GM076267 - NIH; R01GM114061 - NIGMS, NIH; R15GM116032 - NIGMS, NIH; P30 GM103450 - NIGMS, NIH; GM129465 - NIGMS, NIH; GM119426 - NIGMS, NIH; TW009872 - Fogarty International Center, NIH; U01 TW00313 - Fogarty International Center, NIH; R01 CA158275 - National Cancer Institute (NCI), NIH; P01 AG012411 - NIH/National Institute of Aging; Camille and Henry Dreyfus Foundation; Arnold and Mabel Beckman Foundation; Scripps Institution of Oceanography; P01CA125066 - NCI, NIH; 1 R01 AT008088 - National Center for Complementary and Integrative Health; W81XWH-17-1-0256 - Office of the Assistant Secretary of Defense for Health Affairs through the Peer Reviewed Medical Research Program; P30 CA008748 - NCI, NIH, through a Cancer Center Support Grant; California Department of Food and Agriculture Pierce's Disease and Glassy Winged Sharpshooter Board; American Lebanese Syrian Associated Charities (ALSAC); University of Oklahoma Startup funds; 53767-ND1 - ACS PRF; PhRMA Foundation; P30 CA008748 - CCSG NIH; RSG-12-253-01-CDD - American Cancer Society; RSG-13-011-01-CDD - American Cancer Society; CHE-0353662 - National Science Foundation; CHE-1005253 - National Science Foundation; CHE-1725142 - National Science Foundation; Beckman Foundation; Sherman Fairchild Foundation; John Stauffer Charitable Trust; Christian Scholars Foundation)Published versionSupporting documentatio